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Abstract 



1^ I We develop a statistical-mechanical formulation for image restoration and 

C I error-correcting codes. These problems are shown to be equivalent to the 

Q ' Ising spin glass with ferromagnetic bias under random external fields. We 

prove that the quality of restoration/decoding is maximized at a specific set 
of parameter values determined by the source and channel properties. For im- 
age restoration in mean-field system a line of optimal performance is shown 
to exist in the parameter space. These results are illustrated by solving ex- 
^\^ ' actly the infinite-range model. The solutions enable us to determine how 

f^ ■ precisely one should estimate unknown parameters. Monte Carlo simulations 

^» . are carried out to see how far the conclusions from the infinite-range model 

are applicable to the more realistic two-dimensional case in image restoration. 
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I. INTRODUCTION 

Information is usually transmitted through noisy channels. One therefore has to devise 
a method to retrieve the original information from the output of a noisy channel. Let us 
suppose that the original information is represented as a sequence of bits. The idea of error- 
correcting codes |l[] is to introduce redundancy into the bit sequence to be fed into the noisy 
channel, so that this additional information is helpful to retrieve (decode) the original bit 
sequence from the corrupted output of the channel. An example is the parity-check code in 
which the parities of appropriate blocks of bits are sent through the channel in addition to 
the original source sequence. The receiver checks the consistency between the information 
bits and parity bits and takes appropriate actions if an inconsistency is found. 

The problem of image restoration is similar to the error-correcting code, in the sense 
that an image represented by a set of pixels (corresponding to the bit sequence in error- 
correcting codes) is corrupted by noise and the receiver tries to retrieve the original image 
out of the noisy, corrupted one. A major difference is that in the image restoration problem, 
one is usually given only the corrupted image, not other additional redundant information. 
One thus relies on some a priori knowledge about images in general to remove noise. A 
simple instance is the assumption of smoothness in real-world images; for example, one may 
wish to suppress an isolated white pixel among black ones because such a configuration is 
likely to have been caused by noise rather than to have existed in the original image of the 
real world. 

A general strategy common in error- correcting codes and image restoration is to use the 
Bayes formula on the a posteriori probability (posterior) of an output sequence, given the 
input sequence. One then often accepts the sequence (image) which maximizes the posterior 
as the decoded/restored result. This method is called the maximum a posteriori probability 
(MAP) estimate. 

Sourlas pointed out that the problem of error-correcting codes can be written in terms 
of the theory of spin glasses. The idea is first to represent the bit sequence as an Ising spin 
configuration and then to form a set of exchange interactions as the products of appropriate 
sets of spins, which can be considered as a generalization of the Mattis model of spin glasses 
Q. The set of interactions, instead of the spin configuration, is fed into the channel, in 
which noise causes the signs of interactions to flip with some probability. At the receiving 
end, one forms the Ising model Hamiltonian from the corrupted exchange interactions, the 
ground state of which is accepted as the decoded information. This process is equivalent to 
the MAP estimate. Sourlas used this formulation to show that there exists a family of codes 
which are asymptotically error-free and saturate the bound on the code rate (the number of 
information bits divided by the number of transmitted bits) derived by Shannon [Q. 

Rujan [H subsequently proposed to carry out the decoding procedure not at the ground 
state but at a finite temperature corresponding to the Nishimori temperature found in the 
theory of spin glasses . The finite-temperature decoding is effective because the correct 
original bit sequence has a higher energy than the ground state if the exchange interactions 
are corrupted and thus are deviated from the Mattis type. His proposal was supported by one 
of the authors [§] who proved that when the decoding temperature is varied, the average error 
per bit in the decoded sequence becomes smallest at the Nishimori temperature. Sourlas 
used the Bayes formula to rederive the finite-temperature decoding of Rujan under more 



general conditions. A recent development along this line is the contribution by Kabashima 
and Saad [|l3] who used a diluted many-spin interacting model of spin glasses to show that 
this system asymptotically saturates the Shannon bound with the code rate kept finite. 
Analogy with statistical mechanics has also been a useful guide to develop a variety of 



techniques in the image restoration problem |Tl|-p!3|. Most of the efforts, however, have been 



devoted to the development of efficient methods to search for the ground state of appropriate 
statistical mechanical systems (the MAP restoration) using simulated annealing |]TT[, mean- 



field annealing |12| or the cluster variation methods ||13[. An interesting exception is the work 



of Pryce and Bruce B who pointed out that under a certain criterion, the MAP estimate 



is outperformed by the thresholded posterior mean (TPM) estimate |T^, which is actually 
equivalent to a finite-temperature decoding method. 

It is therefore natural to formulate the problems of error-correcting codes and image 
restoration within a unified theoretical framework and apply various techniques developed 
in the theory of disordered systems like spin glasses and the random-field Ising model. In this 
paper, we study the choice of parameters for optimal performance with this approach. For 
illustration, we introduce the infinite-range model, by which it becomes possible to discuss 
analytically the parameter dependence of the performance. 

In Sec. II we give the basic formulation of the problem for the binary symmetric 
and Gaussian channels. The posterior for the output of a noisy channel is interpreted 
as the Boltzmann factor of a statistical-mechanical system, namely the Ising spin glass 
with ferromagnetic bias under random fields. Important quantities in the theory of error- 
correcting codes and image restoration are represented in terms of thermal averages of the 
Ising model at finite temperatures. We derive an upper bound on the overlap between the 
decoded/restored result with the original sequence/image using the statistical-mechanical 
formulation. The problem of image restoration is treated in Sec. Ill in detail where we derive 
a line of optimal performance in mean-field systems, find the exact solution for the infinite- 
range model, and present simulation results for the two-dimensional case. The infinite-range 
model is shown to work as a good guide to the description of qualitative behavior of the 
two-dimensional problem. Explicit examples of images are displayed to clarify what hap- 
pens under various conditions. The problem of error-correcting codes is analyzed in Sec. 
IV. We solve the infinite-range model explicitly using the replica method. Compact ex- 
pressions of the overlap and other order parameters enable us to discuss various aspects of 
error-correcting codes quantitatively. The final section is devoted to discussions. 

II. GENERAL FORMULATION 

Consider an information source which generates a bit sequence represented by a set of 
Ising spins {^i}, where ^j = ±1 and i = 1,---,N, with the source probability Ps{{^i}) 
(the prior). The sequence {^j} is coded as the products of r spins J°...j^ = 61 ' ' '^r ^^ 
appropriately chosen sets of indices {ii---v}- The Sourlas code is equivalent to the 
infinite-range model in which all possible combinations of r sites are chosen from N sites. 
In general we consider several different r's in a single code, such as in the case of the Viterbi 
code which has r = 2 and 3. The problem of image restoration can be regarded as 
a special case of r = 1, in which case Jj corresponds to the state of the ith pixel in the 
corrupted (noisy) image. A "bit" in error-correcting codes should thus be identified with 



a "pixel" in image restoration, a "bit sequence" with an "image", and "decoding" with 
"restoration" whenever necessary in the following arguments. 

When the signal is transmitted through a noisy channel, the output consists of the sets 
{Jii-ir} fo^ various values of r, which are the corrupted versions of Jj°...j^. Two kinds of 
noisy channels are considered in this paper: the binary symmetric channel (BSC) and the 
Gaussian channel (GC). 

A. Binary symmetric channel 

In the binary symmetric channel, the output Jj^.-.j^ is equal to TJi^...i^ with probabilities 
Pr and 1 — Pr respectively, where Pr is the error rate of the BSC for transmission of J^ ...i^- 
The error probabilities of flipping the signal +1 to —1 and —1 to +1 are the same. This 
output probability can be written in a compact form as 



Pon,{{J}m) = n(2cosh/5,)-^^ exp (j2(3rY.Jn-^M^ ■ ■ -^v) , 
where /3r = if the code in consideration does not include the set { Jii-..^} and 



(1) 



ft^lni^ (2) 

otherwise. The second summation in the exponent of Eq. (|I]) extends over an appropriate 
set of the indices (zi, ■ ■ ■ , v), the choice of which determines the type of the code, and A^^ 
is the number of terms appearing in this summation. Note that N^ = ii jSr = 0. Each 
index i may appear in a number of r-spin terms in the exponential expression; the number 
of times of appearance is called the valency Zr ■ 

The procedure of decoding/restoration proceeds as follows. According to the Bayes 
formula and Eq. ([^), the posterior probability that the source sequence is {cr}, given the 
output {J}, is proportional to the Boltzmann factor of an Ising model multiplied by the 
prior: 



P{{a}\{J}) ex exp fe/^^E -^n-v^n ■ --^ir) PsiW})- 



(3) 



Note that we use the symbol {a} for the decoded/restored result which is in general different 
from the original set {^}. 

For simplicity, we restrict ourselves to the case of a single non- vanishing (3r{= Pj) with 
r > 2 and /3i(= (3t)- Following the convention of separating the interaction terms and local 
field terms in statistical mechanics, we write Jj as Tj for r = 1 terms, and Eq. (j^) becomes 

^({^}|{^}, {r}) oc exp (/?j Y. -^n-v^n " " " ^v + /?. E '^^^i) P^iW}), (4) 

Similarly, we will use pj, Pr to represent pr, pi respectively. 

It often happens that the receiver at the end of the noisy channel does not have precise 
information on /3j, /?,- or Pg. One has to estimate these so-called hyperparameters, which is 
one of the major topics in this field (see for example |jl5|). If the receiver estimates (3 for (3j 



and h for Pr, and uses a model (guess) Pm{{(y}) of the source prior P^dO)? then the mean 
of the posterior distribution of ai is equal to the thermal average 

where the Hamiltonian is given by 

PH = -l3^Ji^...i^ai^ ■■■Oi^ -h^TiGi. (6) 

One then regards sgn(o"j) as the ith bit of the decoded/restored information with finite /3 
and h in the finite-temperature process, or with their ratio kept finite when their magnitudes 
approach infinity in the MAP method. 

In the context of image restoration, one often considers patterns with non-trivial struc- 
tures. Therefore they are assumed to be generated by a non-uniform source prior Ps({^}). 
When we do not have any information on the source prior, we have to represent our a 'priori 
knowledge on general images in the model prior P^d^}). A natural choice often used as a 
generic form of the prior is the Boltzmann factor of the ferromagnetic Ising model 

^m({^}) = -777^-^ exp -^ ^ GiOj , (7) 

where Z{[3m) is the partition function at the inverse temperature /?„, {ij) represents interact- 
ing sites, and z is the valency of each site. The summation usually extends over neighboring 
sites on a two-dimensional lattice. This prior is natural because it suppresses different states 
of neighboring sites, enhancing a smooth structure. In this paper we consider priors with 
general connections. 

To proceed further, we have to assume some explicit form of the source prior P^dO)- 
To develop a general theory, we adopt the Boltzmann factor of the Ising model 




i^s({e}) = ^^exp(^^e4|, (8) 

which has the same form as Eq. (|^) but with a different inverse temperature. Thus the 
original images correspond to snapshots of equilibrium Monte Carlo simulations of the fer- 
romagnetic Ising model. 

Comparison of Eqs. (|), (^ and (|^) implies that the Bayes result (^ specifies the 
inverse temperature /3 to /3j, the field strength h to [3^ and the model prior P^ to P^. 
Nevertheless it is useful to keep /3, h and P^ as adjustable parameters and investigate how 
precisely we should estimate the hyperparameters and tune the adjustable parameters for 
optimal decoding/restoration. Our statistical-mechanical formulation is particularly useful 
to investigate this problem both qualitatively and quantitatively. 

The decoding/restoration procedure reduces to the method of Rujan in the situation of 
error-correcting codes {i.e. h = and P, = Pm =const) when (3 = f3j, in which case the 
noise temperature l//9j is equal to the Nishimori temperature, which is known to play an 
interesting role in the theory of spin glasses 0. Pryce and Bruce p| called the same method 



the TPM in the image restoration situation, in which one sets /i = /?,- in the absence of the 
interaction term in Eq. (^. 

The finite-temperature method with appropriate parameters (/? = /?j, h = /?,-, Pm = Ps) 
is known to give the sequence of most probable bits both in error-correcting codes and 
image restoration 0J^,y,|T^. The MAP estimate, on the other hand, chooses the sequence 



which gives the largest value of the posterior (Q), corresponding to the ground state of the 
Ising model. The result of finite-temperature decoding/restoration gives a lower value of the 
posterior (|^) than the MAP result. However, there appear very many states with almost 
the same value of the posterior if we generate finite-temperature states by Eq. (|^), and 
thus after weighted by the number of such similar states, finite-temperature states outweigh 
the MAP counterpart. In other words, if we take into account the entropy effects, the 
finite-temperature method becomes the natural choice. This corresponds to the free-energy 
minimization rather than the energy minimization as in usual statistical-mechanical systems 
at finite temperatures. 

B. Gaussian channel 

In the Gaussian channel, the output Jj^.-.j^ is a Gaussian random variable with appropri- 
ate mean a^Jj^...^^ and variance J^. Hence for a given sequence {^i}, the Gaussian channel 
is given as 

PoutiU}, {r}m) oc exp (-^ E(^n.-v - Jo^^. ' " " U' " ^ E(^^ " <)') (9) 

= exp (-^ E(4...v + Jo) - ^ Ur^ + «^) + ^ E ^n-v^n ■■■u + :^ E^^e.) , (10) 

where, again following the statistical mechanics convention, we have used Jq, J, a, r to 
represent a^, Jr, o-i, Ji respectively. 

Similarity of Eqs. (|) and (p!OD implies that the output probability distribution for BSC 
and GC can be written in the same form, 

^out({J}|{0) = n^'-('^n-v)n^l(^^)exp(/5jE'^n-ven---ev + /5rE^^^^)' (H) 

where /3j = Jq/J'^ and Pr = cl/t'^ for GC. F^ and Fi are functions independent of {ii}. For 
BSC, 

FM,■■■^^ = 2 coll, p J ^^^J^--- - 1) + '^('^^-- + 1)> ' (12) 

^l(^*) = n \o {^(^* - 1) + ^(^* + 1)} • (13) 



For GC, 



2 cosh Pr 



^^^^^"■■"^ ^ T^"""^ ("27^(^^-^^ + -^^^ ) ' (^^^ 



F,{n) = -^^e.p^- — irt + a^)). (15) 



C. Overlap 

The most important quantity in the present problem is the overlap of the de- 
coded/restored bit sgn(cri) and the original bit ^i averaged over the output probability. 
We may express this overlap as 

l[ldJl[JdrPont{{J},{r}m) e.sgn(a,). (16) 



This expression (|T6]) should be further averaged over the possible sequences of source bits 



represented by the prior -Psd^C})- The final expression of the overlap M is then 

M{P,h,P^) =Y.l[J dJUj dTP,m)P,U{J},{T}m)^,sgn{a,). (17) 

The dependence of M on (3,h and Pm exists in the thermal average (ctj). The average of 
any other quantity f{a) is calculated similarly: 

[if)] = T.Uj dJUj drPsm)Pou.{{J},{r}m)^^^^^j^^^ (18) 

The outer brackets [■ ■ ■] in Eq. (|l^) denote the averages over {^}, {J} and {r} with the 
weight PsPont- 

The following inequality on the overlap is very useful in discussions on the decod- 
ing/restoration performance, and the proof is outlined in Appendix A: 

M{(3,h,Pm)<M{(3j,(3,,P,). (19) 

This inequality means that the overlap becomes largest when P = Pj, h = P^ and Pm = Ps- 
In this sense the performance of the MAP corresponding to the limit P,h ^ oo cannot 
exceed that of the finite-temperature decoding/restoration at /3j and Pr- 

This inequality was known in error- correcting codes P,^, and equivalent statements were 
given also in the image restoration problem WM although not in the explicit form given here. 
One of the contributions of the present paper is that we have generalized the inequalities to 
the case where two different types of terms are present, one with r > 2 and the other with 
r = 1. Similar inequalities can be easily derived for the more general case of several different 
r's in Eq. (|l]). 

III. IMAGE RESTORATION 

A. Line of optimal performance in mean-field models 

In conventional image restoration problems, the output of the transmission channel only 
consists of the set of pixels {r,} (corrupted image in the usual sense), but not the set of 
exchange interactions {Jij} (the corrupted version of { J°- = C,iC,j})- In this case the following 
inequality applies to images with extensive valency. Mean-field results are exact when each 
pixel interacts extensively with other pixels in the source and model prior distributions (H) 



and (^. These include the infinite-range model, in which all pixels interact with each other, 
its randomly diluted version with infinite valency in the thermodynamic limit (A^ — *> oo), 
or finite-dimensional models with long-range interactions. Mean-field approximations also 
work well for finite but large valencies when the temperature is not too low. The following 
inequality derived in Appendix B is useful in finding the optimal restoration performance, 

M{h,PJ<M((3,s,^^^s] (20) 

\ m ) 

for arbitrary values of s > 0. Here mo and m are the (self-averaging) thermal averages of 
the source pixels ^j and model pixels Uj respectively, for those sites i which interact with a 
given site, say site 1, which however is removed from the thermal process. Note that the 
derivation only makes use of the self-averaging nature of the mean-field quantities, and does 
not rely on any connection topology of the sites (except for the mean-field requirement). No 
particular techniques such as the replica method are employed. Hence the inequality applies 
to mean-field systems in general. 

Since TTio and m are functions of /3s, (3ra and /i, Eq. (^) defines the line of optimal 
performance in the space of h and (3m'- 



h 


Pmm{(3s,(3m, 

PsmoiPs) 


h) 


= s. 








(21) 


--/3r, 


Pm 


= Ps 


and 


m = 


= mo, 


Eq. 


m 


reduces to the 


point of 



In particular, when s = 1, h 
optimal performance predicted by Eq. (0). On the other hand, if the field h is different 
from j3r, then the source and model temperatures have to be rescaled by the magnetization 
of the respective systems. When s — > cxd, we obtain the zero-temperature restoration, i.e. 
the MAP estimate. Hence the MAP estimate is also optimal, provided that the correct ratio 
of h/Prn is used (although this choice can only be determined iteratively) . 

B. The infinite-range model 

Let us now suppose that we are given the corrupted version of the set of pixels {tj} 
and, in addition, the exchange interactions {Jij}, the latter being the corrupted version 
of the Mattis-type interactions {CiCj}- The additional contribution from the exchange term 
enables us to formulate the image restoration and error-correcting code problems on the same 
footing and at the same time investigate the extent to which the additional information {Jij} 
enhances the performance. The restoration process is carried out at the inverse temperature 
P and the field h using Eqs. (^) and (||) with r = 2. Though the inequalities ([T9|) and (pOD 



give a bound on the overlap M, they do not reveal the explicit dependence of M on the 
parameters /3, h, Prn and Pr, which is necessary for studying the tolerance of the restoration 
results against uncertainties in the estimation of the hyperparameters. The infinite-range 
model serves as a useful test ground for this purpose. Of course the infinite-range model 
is not useful for restoration of a real two-dimensional image since all pixels are neighbors 
of each other and hence the spatial structure is ignored. However, we may reasonably 
expect from experience in statistical mechanics of many-body systems that the behavior of 
macroscopic quantities (such as the overlap M) of realistic problems are at least qualitatively 
well predicted by the infinite-range model. 
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We therefore suppose that the summation in the prior (H) extends over all possible pairs 
of sites 



Psm) 



Zi(3^ 



expUTfE^^O 



2N^ 



(22) 



ij^j 



and similarly for the model prior Pm- We also assume that the two-body exchange interac- 
tions for all pairs of sites are included in the given information, or equivalently Eq. (^) with 
r = 2 and the summation extending over all pairs of sites, and that the channel is Gaussian. 
We thus have to evaluate the following averaged replicated partition function 






exp 






2J2 f-l'"'" A^"^'"^-' 



X 



X 



n 

i 
1 



dTi 



V2ttt'' 



exp (-^E(^i-<i)M 



Ps 



Zif3,)''''^\2N^^'^' 



iy^j 



Pn 



xEexp|/3E^^.E^.X + ^EE<^; + ^E^^E^.^ 



(23) 



«=i 



The standard replica calculation with the replica symmetric ansatz |T^ leads to the expres- 
sions of the order parameters: 

[6] =mQ = tanh/^^mo 
[(^i)] = m = 1^ ^ Y. e^'"""^ f DxtanhU 



[6(^i)] =t 



{^^? 



2 cosh /3s mo .^i 
1 



2 cosh /3s mo 



Y, ^e^="^o« f DxtanhU 



e=±i 



1 



y e'^^'"o« Dxtanh^U, 
2 cosh Psmo /^-^ J 



where 



U = (f3'^J\ + T^h'^) ^''^ X + (3mm + {ah + /3 Jot)^. 



The overlap is a function of these order parameters 

1 Y ^e'^'"^"^ f DxsgnU 



[^isgn{ai)] = M 



2 cosh /3s mo 



(24) 



(25) 



(26) 



?=±i 



An example of the dependence of M on the model prior temperature T^ = /3~^ is shown 
in Fig. [^ for conventional image restoration (without the exchange term, (3 = 0). The 
parameters are Tg = 0.9 and a = r = 1. The usual practice in image restoration is to use a 
Hamiltonian with a fixed ratio of h/ (3mi and then to use (3m as an adjustable parameter for 



simulated annealing. Hence we consider the behavior as a function of f3m when h/f3m is kept 
constant to the optimal value (3t/Ps (the curve marked 'Opt /i'), to 0.9 times the optimal 
value ('Opt*0.9'), or to 1.1 times the optimal value ('Opt*l.l'). In the curve marked '/i = 1', 
h itself is kept constant to 1. The ground-state limit T^ -^ gives the MAP restoration. 
The maximum is at Tm = 0.9(= Tg) for the optimal choice oi h = 1(= PrPm/Ps), as predicted 
by Eq. (|19D. This figure indicates that one does not have to approach the zero temperature 
limit as in MAP in the search of the best restored image by such a process as simulated 
annealing. 

For the curves Opt*0.9 and Opt*l.l, the location of the largest M is not at T^ = 0.9. 
However, the maximum value coincides with the best value as predicted by Eq. (|20|). This 
fact can easily be verified by differentiating Eq. (p6D with respect to Pm- The optimal 
parameter is (3m = Ps^nQh/Pr^n, agreeing with Eq. (pT]). 

The line of optimal performance in the space of T^, and h is obtained by combining Eqs. 
(0) and (g§, yielding 

When h ^ 0, Tjn approaches the limit (rm)max = '^ + 0'Pt/Ps- This is the temperature above 
which the maximum overlap cannot be achieved. 

In the low temperature limit, h —* oo and the ratio h/ P^n approaches a constant 

'- ^ = /^ E i (1 + -of) erf (^ (^ + <)) . (28) 

Hence the maximum overlap is achievable for any temperature below (Tm)max- The zero- 
temperature restoration (the MAP estimate) is potentially as optimal as the finite temper- 
ature procedure determined by Eq. (|l^) in mean-field systems, although it can only be 
achieved at the correct ratio h/ P^ given by Eq. (p^). 

Figure ^ shows the line of optimal performance for the parameters used in Fig. |l|. The 
hyperbolas Tmh = 0.9, 0.81 and 0.99 correspond to the lines of operation Opt h, Opt*0.9 
and Opt*l.l of Fig. |I] respectively. Where they intersect the line of optimal performance, 
the overlap reaches a maximum. 

In realistic image restoration, the precision in the estimation of hyperparameters is an 
important issue. One can see that the three lines of operation follow the general trend of 
the optimal curve. Hence they are much more error-tolerant than other curves, say, h = 1. 
Furthermore, if a line of operation intersects the line of optimal performance with a small 
angle between the tangents, then the overlap M is very near to its optimum for a wide 
range of parameters along the line of operation, and the procedure has a high tolerance for 
parameter uncertainties. Among the three lines of operation in Fig. Q, Tmh = 0.99 has the 



highest tolerance. In fact, if one uses T^h = 1.0267 according to Eq. (p8|), then it has the 
widest range of tolerance in the low temperature region. 

Figure |^ shows the effects of introducing the exchange term. It depicts M as a function 
of the inverse exchange temperature P in Eq. ( ^5] ) with the other parameters set to the 
optimal values Tg = T^, = 0.9, a = h = 1.0 and with Jq = 2.0 in the unit J = r = 1. The 
axis P = corresponds to the optimum point T^ = 0.9 of Fig. |l|. The introduction of the 
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exchange term is seen to sharply improve the performance. The maximum of M is located 
at /? = 2.0(= Jo) as required, and M stays close to the maximum value beyond /? = 2.0. 
The combination of the ideas of error- correcting code (the /?-term) and image restoration 
(the 13m and h terms) leads to a remarkable improvement in the quality of restored image. 
Two remarks are in order in relation to Fig. ^. First, the amount of information conveyed 
by the set {Jij} may seem exceedingly large compared to that by {tj} because the number 
of elements in the former set is A^(A^ — l)/2 while it is A^ in the latter. This fact may be 
mistaken as the reason of the improved result in Fig. |^ for finite j3. However, since each 
Jjj(~ 0{l/yN)) is much smaller in magnitude than rj(~ 0(1)), the contribution of each Jij 
is very small. Such a situation is characteristic of the infinite-range model. The equivalent 
situation in the finite-dimensional case is that the number of exchange interactions is of 
the same order as that of sites. For example, there are 2N nearest neighbor interactions 
for N sites on the square lattice. Therefore, the increase in the amount of information by 
the introduction of the set {Jij} should be of order unity, not infinitely large. Secondly, as 
the exchange term is seen to increase the overlap very sharply, even the information from a 
fraction of the exchange interactions may be useful to improve the restoration result. For 
example, one may choose a small fraction of pairs of sites (either randomly or not) and use 
the corrupted, noisy version of these exchange interactions to restore the image to obtain a 
better result. This method should be useful when the bandwidth (the amount of information 
to be carried by the channel) is limited. 

C. Simulations 

It is difficult to investigate the more realistic case of two-dimensional images by analytical 
methods. We therefore have carried out Monte Carlo simulations to confirm the qualitative 
pictures obtained by the exact solution of the infinite-range model. To generate the source 
image, we have used the prior (|) with Tg = 2.15 which is slightly lower than the critical 
point 2.269 of the two-dimensional Ising model on the square lattice. The error probability 
was set to pj = Pr = 0.1 for BSC, corresponding to (3j = (3r = 1.0986 by Eq. (|). Averages 
over 5 samples (Fig. ^, size 400 x 400) or 10 samples (Fig. ^, size 100 x 100) were taken at 
each data point. 

Figure ^ shows the overlap M as a function of T^ when /3 = and h is chosen so 
that h/ (3m is fixed to the optimum value Pr/Pg. The overlap should have a maximum at 
Tm = Tg = 2.15 in Fig. ^ according to Eq. (|Typ although it is not very clearly seen due 
to statistical uncertainties. It is at least true that M does not change significantly below 
Tm = 2.15. It is therefore unnecessary to lower the temperature than Tg = 2.15 to obtain a 
better result. 

The effects of exchange interactions have been taken into account in Fig. |^ where Tm 
and h are fixed to the optimal values 2.15(= T,) and 1.0986(=/5t-) respectively. The overlap 
is seen to increase quite significantly as a function of j3. The axis j3 = corresponds to 
the optimal point of Tm = 2.15 in Fig. ^. The overlap reaches its maximum at around 
P = Pj = 1.0986 as it should and decreases slowly as (3 is further increased. 

Respectively, Figs. |^ and ^ are quahtatively similar to Opt h in Fig. |I|, and Fig. ^, for the 
infinite-range model, implying the usefulness of the infinite-range model as an approximation 
of the two-dimensional problem. 
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Let us show an explicit example of the actual image restoration. Figure ^ represents 
the situations of Figs. ^ and ^ with the size 100 x 100. We have generated a pattern by 
the prior (H) with Tg = 2.15 to obtain (a) and have added noise with probability Pr = 0.1, 
resulting in (b) [|l^]. To obtain the restored images (c) and (d) only the corrupted image 
(b) was used without extra information on exchange interactions {jS = 0). Restoration was 
tried at temperatures Tm = 0.5 for (c) resulting in M = 0.888 and at T^ = 2.15 for (d) with 
M = 0.892. It is clearly recognized that the optimal temperature Tm = 2.15 (d) has a better 
restored image than (c). The low-temperature process (c) suppresses small structures which 
were actually present in the original image. The low temperature result is close to the MAP 
estimate (T^ = 0) which would further suppress small structures. It should be noticed that 
the difference in M in these two restored results (c) and (d) is very small (which is also seen 
in Fig. ^) but the intuitive impressions on similarity to the original image (a) are rather 
different. The reason is that the small structures do not contribute significantly to the value 
of M although such structures have strong influence on intuitive impressions. Therefore we 
should keep in mind that the overlap M alone does not represent all aspects of the quality 
of restored images. 

We next consider the effects of the additional information of exchange interactions among 
nearest neighbors. The same corrupted image (b) has been used to obtain the restored image 
(e). The parameters T^, h and P were fixed to the optimal values 2.15(= Tg), 1.0986(= P^) 
and 1.0986(= (3j) respectively, resulting in an overlap of M = 0.986. Fine structures are 
remarkably well restored in the result (e). Thus the additional information of exchange 
interactions is very effective to restore images faithfully. 

IV. ERROR-CORRECTING CODES 

The infinite-range model has the same significance in error-correcting codes as in the 
image restoration problem; namely, an exactly solvable model which describes more realistic 
situations at least qualitatively. The difference is that we consider a general value of r in 
error-correcting codes, instead of only r = 2 in the case of image restoration. We therefore 
calculate the overlap M and related quantities explicitly assuming that the set {ii, • • • , v} 
in Eq. (^ extends over all possible combinations of indices. 

We consider the Gaussian channel, and the source and model distributions are both 
assumed to be uniform, Pg = Pm = 2~^ , as is customary in the theory of error-correcting 
codes [p|,P,pHlO| . /^From Eqs. ( |T7| ) and (|]), the overlap is given by 

M{P,h) 







\2 



27rr)^ ' V 2r2 

xe. sgn /^ I>^exp (/3E^n...v^n -v ._^ ■^^^, j _ (29) 

V E exp [(3 Y. Jh-ir(^h ■■■(^ir + hY. ncTi) ) 
The normalizations of J and jo are different from Eq. (0) and follow the convention of the 



infinite-range model of spin glasses so that the limit r ^ oo yields meaningful results [[16 |. 
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We may change the signs of integration variables in Eq. (^) appropriately {Ji^-i^ —>■ 
Jii-irik ■ ■ 'iiri 'Ti ~^ 'Tiiii ^i ~^ ^iii) which allows US to drop ^'s from the integrand (the 
ferromagnetic gauge). Then the problem becomes the standard mean-field theory of spin 
glasses with r-spin interactions under external random fields, and we can apply the well- 



established replica method |T^. Standard replica calculations under the replica-symmetric 
(RS) ansatz lead to the following set of equations of state for the spin glass order parameter 
g, ferromagnetic order parameter m and the overlap M: 



q= Dx tanh^ G (30) 

m = DxtanhG (31) 



M = DxsgnG (32) 



where Dx is the Gaussian measure and 



/ f}2i2r-l \l/2 

G = ^ Y + T^h^ X + /?jor m"-i + ah. (33) 

The corresponding free energy is 

/rs = -Tlog2 - ^ + ^-{l - r)q'' + ^rg"-i + jo(r - l)m" - f Dx log cosh G. (34) 
The present system with jo = is known to have a spin glass phase with a single- 



step replica-symmetry breaking (IRSB) at a low temperature when r > 3 0]. This spin 
glass phase with IRSB is replaced by a full-step replica-symmetry breaking at a still lower 
temperature. It is therefore necessary to study replica-symmetry breaking solutions following 



Refs. IT^JT^ . The stability condition of the RS solution (the AT line) is found to be 

^ > Dxsech'^G. (35) 



r{r — 1) J2 
The free energy with IRSB {h = for simplicity) is 

/iRSB = -Tlog2 - ^ + ^Xo(l - r)gS + ^(1 - a:o)(l - r)g[ + ^rg[-i 



+jo(r - l)m'' f Dulog f Dv cosh^" Gi (36) 

Xn J J 



where 



Gi = u^'^pUYo-' + v^'^^qr' - ql-^ + ^wnf-\ (37) 

The self-consistent equations for the order parameters are obtained by extremization of Eq. 
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go 



91 



m 



I 



Du 



Du 



Du 



f Dvcosh'^'GitanhGi 

I Dv cosh^° Gi tanh^ d 

f Dv cosh^° Gi 
f Dvcosh''''GitanhGi 
Dv cosh^o Gi 



(38) 



(39) 



(40) 



We do not write out the explicit form for the equation of xq because the formula is not very 
instructive. The AT stability of this IRSB solution is 



2-rrp2 



2gf-"T 



r[r 



1)J2 



> 



Du 



Dt;cosh^°-*Gi 
Dvcosh'^^Gi 



(41) 



The phase diagram in the case of r = 3 and h = is shown in Fig. ^ Retrieval is 
not possible unless the ferromagnetic phase is at least locally stable. The hatched region 
satisfies this condition. The ferromagnetic phase is stable in the replica symmetric ansatz 
for sufficiently strong bias jo and high temperature T. For T/J above and below 0.651, it is 
respectively replaced by the paramagnetic and spin glass phases through first-order phase 
transitions when jo decreases. These three phases coexist at the triple point (TP). The 
ferromagnetic phase remains metastable down to the spinodal line shown as a dotted curve. 
The replica symmetric solution of the ferromagnetic phase becomes unstable below the AT 
line (^) shown by the dash-dotted curve. 

Also shown in Fig. are the spin glass phases, which exist at lower values of the bias 
jo- Spin glass with a single-step replica symmetry breaking is stable for T/J between 0.651 
and 0.240. At lower temperatures, it is replaced by a full replica symmetry breaking spin 
glass phase. 

Investigation of the properties of the mixed phase M (such as distinction between IRSB 
and full RSB), as well as the deatils of the spin glass phase, are interesting future problems, 
which we do not pursue here since they are not directly relevant to our problem of error- 
correcting codes around the optimum temperature T = J^/2jo shown dashed in Fig. ^ We 
have not shown the structure of the phase diagram at very low temperatures for this reason. 

Figure || shows the dependence of the overlap M on the decoding temperature T = (3~^ 
with r = 3 and jo/ J = 0.77. The line jo/ J = 0.77 lies slightly to the right of the triple point 
TP in Fig. |^. The maximum performance is achieved at T = J^/2jo = 0.649. This result 
is consistent with the argument in Sec. II C: if we repeat the proof of Eq. (|19]) with the 
external field neglected and Pg = Pm = 2~^ , we obtain the inequality M(/5) < M(2jo/J^). 

The optimum condition T = J^/2jo coincides with the Nishimori line shown dashed in 
the phase diagram (Fig. |^). This curve crosses the phase boundary and the spinodal line 
at the points where jo takes the smallest values in the ferromagnetic stable and metastable 
phases respectively (marked by black and white circles). It can be shown that the spinodal 
line, the Nishimori line and the AT line are concurrent for any values of r. Hence the AT 
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line terminates at the triple point. Since the Nishimori line lies in the replica symmetric 
phase, the replica symmetric argument would be sufficient to clarify the behavior of the 
overlap around its maximum. The lower temperature properties, including the possibility 
of a reentrant spin glass phase, may be affected by replica symmetry breaking. 

Figure ^ shows the dependence of M on the random field strength h at the optimal 
temperature T = 0.649 with r = 3, j^/ J = 0.77 and a = 1. The axis h = corresponds to 
the conventional Sourlas code without the field term, which is the maximum point in Fig. 
^. It is observed that the overlap M increases sharply as the field is introduced, reaching 
the maximum at /i = 1, in agreement with the theoretical prediction h^pt = a/r"^. 

V. DISCUSSIONS 

We have formulated the problems of image restoration and error-correcting codes in a 
unified framework using statistical mechanics. We have derived an upper bound on the 
overlap M between the restored/decoded image/sequence and the original image/sequence. 
The maximum of M is achieved when the restoring/decoding temperature and field strength 
match the corresponding temperature and field strength characteristic of the source and 
channel properties. This result comes as a natural generalization of the previously known 
inequalities for image restoration W^ and error- correcting codes [H. The formulation and 



the proof of the inequality have a formal similarity to the theory of spin glasses, in particular 
the one using gauge symmetry 0. One should note however that we have not used gauge 
symmetry in the present paper. The variables {^j} in Eq. ([T8|) , playing a central role in 
the spin glass theory [j^, come naturally in the present problem whereas they emerged as a 
result of gauge transformation in the spin glass theory. 

The infinite-range model has been solved exactly both in the image restoration and 
error-correcting code situations. The results made it possible to reveal the dependence of 
the overlap M on various parameters. Simulations for image restoration have confirmed that 
the results for the infinite-range model remain qualitatively valid in two dimensions. 

For image restoration in mean-field systems, we have found a line of optimal perfor- 
mance along which the overlap M takes the same maximum value. The line contains the 
point of optimal performance predicted by the inequality (|19]), but extends also to the zero- 
temperature limit. This indicates that optimal (or quasi-optimal) performance is far more 
accessible than previously thought. It remains to study the extent to which the picture is 
applicable to finite-dimensional systems where the mean-field theory is only approximate. 
In this respect, it is interesting to note that a ridge of nearly optimal overlap has already 
been observed in early literature, such as Fig. 2 of |T^ and Fig. 8 of 0]. Naturally, one is 
led to expect that a narrow but extended region of optimal (or near-optimal) performance 
spans the parameter space. 

By comparing the optimal line and the operation lines on which h/ j3m is kept constant, 
we have studied the tolerance towards uncertainties in parameter estimation. Apparently 
the zero-temperature restoration (the MAP estimate) is most robust. Furthermore, if the 
MAP estimate is approached by simulated annealing, it may be more effective to consider 
rescaling the field strength while lowering the temperature at the same time. 

However, we have a few remarks of caution about the MAP estimate, (a) The zero- 
temperature restoration is optimal only when the correct ratio h/ (3m is used, which can 
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only be found self-consistently in realistic situations; if the incorrect ratio is used, the per- 
formance will be sub-optimal, (b) The existence of the line of optimal performance in 
finite-dimensional systems remains an open issue. Simulations in two dimensions seem to 
show that the MAP estimate is sub-optimal, although most likely it is still nearly optimal. 
On the other hand, the optimal point predicted by Eq. (|T9D is guaranteed to be the best in 
general cases, (c) The present result applies to the equilibrium state of the system, and the 
dynamics remains an open issue. It may happen that the approach to equilibrium at a low 
temperature is much slower, or is more prone to being trapped by local minima. 

We have also considered the inclusion of exchange interactions as extra information in 
image restoration. Explicit examples of images in two dimensions show that the fine struc- 
tures are remarkably well restored. We remark that the exchange interactions have some 
similarities with "line processes" , which has been proposed to improve the quality of images 



111 . If the line variables were quenched, they are equivalent to binary and multiple inter- 
actions among neighboring sites. However, a major difference is that the line variables are 
dynamical in the process of image restoration, whereas the exchange interactions considered 
here are quenched. 

A comment is in order on the amount of information carried by the channel of the infinite- 
range model. The signal amplitude of the exchange term in Eq. (EDf) is JQr\/N'''~^. The 
channel noise causes fluctuations in the output with the standard deviation J{r\/N^'~^Y^'^, 
which is much larger than the signal itself when A^ ^ 1. This corresponds to an extremely 
low signal-to-noise ratio, yet the output still contains significant information of the original 
message. This demonstrates the power of the infinite-range decoding scheme in extremely 
noisy situations, although in practice such extremes do not occur frequently. 



Finally, we mention briefly the idea of selective freezing ||20| . The Ising spins keep moving 
under thermal agitation when we employ the process of finite-temperature restoration. Some 
spins have smaller thermal fluctuations than the others, resulting in larger local magnetic 
moments. It may thus be interesting to fix (freeze) those relatively stable spins to ±1 
according to the sign of (cxj) and repeat the finite-temperature decoding/restoration process 
for the other less stable degrees of freedom. We call this idea the selective freezing, which 
turns out to enhance tolerance against uncertainties in parameter estimation. The details 



will be presented in a forthcoming paper 21 
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APPENDIX A: THE INEQUALITY FOR GENERAL DECODING AND 

RESTORATION 

To prove the inequality ([T9|), we first note that the argument of the summation in the 
definition (|l^) is bounded by its absolute value: 

M{(3,h,Pj 
= E n / dJMJ) n / ^^^i(^) exp {Pj E Jn-Mi ■ ■ ■ a + /5r E ^^^^) Ps{m)^^ sgn(a,) 



<]lJdJFriJ)Y[JdrF,{r 



^ 6 exp {Pj Y: \-iM^ ■ ■ ■ ^v + /5r E ^i^^) Ps{{i)) 



(Al) 



where |sgn(crj)| has been replaced with 1. Using the identity |x| = xsgnx, we get 

<En/^'^^'-(^)n/^^^iw(Ee.exp(/3j^j.,....,,e.,---ev + /?rE^^e.)^.({0)] 



X sgn 






(A2) 



Thus the right hand side can be interpreted as the average of the product of ^j and sgn(cri) 
at the optimal parameter values /3 = /3j, h = [3r and Pm = Ps, yielding Eq. (pJl). 



APPENDIX B: THE INEQUALITY FOR MEAN-FIELD IMAGE RESTORATION 

To derive the inequality (PU[), we start with the definition (|T^. Substituting Eqs. ([TT]), 
(I) and (1^) we obtain, for i = 1 in the average, 



M{h,(3m) 



^n/^^^^i(^^)E6exp(^^a, + /5.E^«ei) 

X sgn ^ (Ti exp -^ ^ GiOj + h^^ nai 



(Bl) 



In the exponential argument of the Boltzmann factor containing {^}, ^i only appears in the 
expression z~^(5s H{ij) ii^j + PtTi^i- Hence if we multiply and divide this expression by the 
partition function of {^i} excluding site 1, we have 

E ^1 <^^P — E ^iO + /^r E ^i-^i " E exp — ^ 60+/^^E '^i^i 
ii V ^ m i ) ir V ^ {ij) i 



^ r/i exp I I — ^ r]j + /^^n | r/i 



[B2) 



H{PsM\^ 
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where (■ ■ ■)H(Ps,liT)\^ represents the thermal average taken over the Hamiltonian with inverse 
temperature f3s and random field strength /3^, excluding site 1. Similar arguments can be 
applied to the argument of the sign function in ( |B1| ), yielding 

2/sinh(^E^^+/5^^i)) sgn/sinh(^E^J + ^^i)) • (B3) 

For mean-field systems, J2{ij) Vj ^^^ ^(ii> '^j ^^^ self-averaging quantities [Q, and the ther- 
mal average of the hyperbolic sine functions can be replaced by a single function of the 
thermal averaged argument. Thus M{h,Prn) reduces to 

2 sinh -^ E(^i)/^(/3.,/3.)\^ + /^-^i sgnsinh ^ E(^i)^(/3m,/»)\i ^hrA. (B4) 

For mean-field systems with large valency, the averaging over the neighbors of site 1 reduces 
to the disordered average. Consider [(%)^^], which is the thermal and disordered average of 
•x]^ taken over the Hamiltonian H[f3s,f3T 






[^^^■)^^] - Z(A 






After canceling terms in the denominator and numerator, we arrive at 



(B5) 



—- W / rf7^Fi(ri)E ^Vj exp — E ^Vi^k + /?tE ^^^^^ > (^6) 

^'') i^l •' Vi V ^ (ik) i J 



which reduces to [(■Cj)^^] = ""^o? namely the magnetization in the prior distribution. Similarly, 
[(crj)"^^] = m, which is the magnetization in the model distribution. 

Substituting these results, and using the normalization of the probability 
Ec^'Ps({e.}\')^({T-i}\^|{e^}\'), Eq. (0) reduces to 

Z{(3s 

The rest of the proof is similar to Appendix A. Noting that the integrand of r is bounded 
by its absolute value, we have 



2Z(3 l^"*" r 
M{h,P^)= ^^J J drF, (r) sinh (/5,mo + /5,r) sgn (/5^m + hr) . (B7) 



2Z{P,)\' 



M{h,(3J< ^'-y J drFi (r) sinh (/3,mo + /3.r) sgn (/3,mo + /3.r) . (B8) 



The right hand side is the value of M(/i, [3m) when Eq. (^) is satisfied, since in this case, 
sgn(/35mo + /^^r) = sgn(^mm + hr). 
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FIGURES 

FIG. 1. The overlap as a function of the restoration temperature Tm in the infinite-range model. 
The random-field strength h is chosen to he h = jSrPm/Ps (Opt h), h = 0.9l3rl3m/Ps (Opt*0.9), 
h = l.lprPm/Ps (Optn.l), or h = l. 

FIG. 2. The line of optimal performance in the space of the random field strength h and the 
restoration temperature Tm in the infinite-range model for the parameters used in Fig. ||. The 
three lines of operations (Opt h, Opt*0.9 and Opt*l.l) are shown for comparison. 

FIG. 3. The overlap as a function of the parameter /? in the infinite-range model. The center 
of the channel output distribution is Jq = 2.0 and the other restoration parameters are chosen to 
be the optimal values T^ = Ts{= 0.9) and h = a(= 1). 

FIG. 4. The overlap as a function of the restoration temperature on the square lattice obtained 
by simulations. The source temperature is Tg = 2.15, the error rate Pr = 0.1, and no exchange 
interactions (/3 = 0). 

FIG. 5. The overlap as a function of (3 for the square lattice when T^ = Tg = 2.15 and 
h = I3r = 1.0986. 

FIG. 6. Examples of image restoration. The original image is (a), and the image corrupted 
by the noise {jpr = 0.1) is (b). The restored images are (c) {Tm = 0.5), (d) {Tm = 2.15), and (e) 
(with the exchange term). 

FIG. 7. The phase diagram of the r = 3 system. Message retrieval is possible in the stable and 
metastable ferromagnetic phases shown hatched. 

FIG. 8. The overlap M as a function of the decoding temperature in the Sourlas code. 
Three-body interactions r = 3 are considered and the center of the channel output distribution is 
Jq/ J = 0.77. The field term is /i = 0. The replica-symmetric solution shown here is unstable below 
the AT line at T = 0.43 (shown dotted) although we do not expect a significant deviation in the 
temperature range T > 0.40. 

FIG. 9. The overlap as a function of the field strength h. The conventional statisti- 
cal-mechanical formulation of error-correcting codes (Sourlas code) corresponds to the axis /i = 0. 
The parameters are r = 3, jo/ J = 0.77, a = 1 and T = 0.649. 
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